Virtual workstation

ABSTRACT

In accordance with the present invention there is provided devices and methods for controlling a microprocessor controlled device in a virtual environment through the use of at least one sensor disposed at least one user&#39;s finger and a scanner, said sensor and scanner being utilized to input position data, the method comprising the steps of: loading an operating system in a computing environment; displaying a virtual keyboard and a virtual pointing device on a display device; initializing coordinates defining individual keys of said keyboard; initializing coordinates defining a location of said pointing device in relation to the keyboard; monitoring the position of at least one sensor disposed on a user&#39;s finger; determining if sensor movement correlates to depression of a key on the keyboard and providing feedback to the user to indicate that a key was depressed on the keyboard; transmitting data correlating to the depression of the key; and returning to the monitoring step.

PRIORITY CLAIM

The present invention claims priority to U.S. Provisional Patentapplication Ser. No. 60/424,557 filed Nov. 6, 2002, the entirety ofwhich is hereby incorporated by reference.

TECHNICAL FIELD

The present invention is generally related to methods and devices forproducing a virtual computing environment. More specifically, thepresent invention is related to methods and devices that in combinationmimic the functionality and behavior of a physical computer system in avirtual environment. In particular the methods and devices in accordancewith the present invention includes a portable computing system whereinthe system is configured to provide the user with a virtual hardwareenvironment that may be utilized for computing, gaming or other uses.

BACKGROUND OF THE INVENTION

With the advent of the modern computer input devices were invented toallow for the input of data, for example, early computer systemsutilized punch cards to input data into the computer's memory. Althoughpunch cards were effective at inputting data, a more simplified devicewas necessary, thus the modern keyboard was developed.

One of the most important factors contributing to the effective use of acomputer is the interface between the computer and a person using it.Unquestionably the most popular computer interface device is thekeyboard, which has a plurality of depressible keys each correspondingto a particular alphanumeric character, symbol, or computer function.While computer keyboards are widely accepted and quite suitable in manysituations, keyboards are not always the most efficient, convenient, oreasy to use devices.

A drawback of computer keyboards is that they include up to 110individually marked keys mounted to a base with as many switches. All ofthese components must be produced and assembled, which accounts forconsiderable expense. Since keyboards are mechanical, are also moreprone to failure than solid-state devices, additionally, due to thelikelihood of failure, broken keyboards additionally present disposalproblems. Further, the conventional keyboard cannot be quickly changedto a new keyboard layout, such as might be desired by those who havelearned a keyboard layout other than the somewhat inefficient buttraditional QWERTY layout.

Another drawback of computer keyboards is that they are built generallyin one size for all users. As a result, users with relatively small orlarge hands must adapt to a keyboard size that is not optimal for theirparticular hand size. A person with smaller hands must stretch furtherto strike a key some distance from a home row of keys, whereas a personwith larger hands will have a harder time accurately striking anydesired key. Keyboard size that is optimized for a particular use maylead to decreased hand fatigue. However, keyboard manufacturers havedetermined an ergonomically acceptable compromise, which is a compromisenevertheless. Since keyboards are produced having only one size forces auser to type with his hands close together in an unnatural manner. Ithas been found that so called “split” keyboards, which are split into aseparate keyboard for each hand, are more comfortable for the user andproduce a slightly faster typing speed as a result. Additionally, ascomputers become more common in the workplace, a greater number ofinjuries have been reported due to utilizing a keyboard.

There have been attempts by various manufacturers to address theproblems associated with mechanical keyboards. One such example isdescribed in U.S. Pat. No. 5,581,484, wherein there is described afinger mounted computer input device. The finger-mounted device utilizesa series of pressure sensors to determine a users hand movement whichthen corresponds to a key on a keyboard. A problem with this type ofsystem is that the user must still physically interact with a surface togenerate a signal. Additionally, the sensors are usually disposed on aglove, wherein the user wears the glove to utilize the system. A problemassociated with glove-based systems is that the material from which theglove has been fabricated has a fatigue life, and therefore willeventually wear out from prolonged usage. Additionally, a user mayexperience discomfort from using these types of gloves in that they mayperspire inside the glove. The perspiration may further lead todegradation of the glove.

Another example of a virtual keyboard are produced by www.vkb.co.il,www.canasta.com, and www.virtualdevices.net. These types of keyboardsutilize an infrared projection system, wherein a keyboard is projectedonto a surface and a sensor detects the position of a finger on top ofthe projected keyboard image. A problem with they types of keyboards isthat you can only utilize the system on a smooth clean non-transparentsteady surface, additionally, if you rest your hands within theprojected keyboard the sensor may interpret this motion as keystrokes,thereby resulting in errors. Further still, since the keyboard isprojected onto the surface, the user may experience light interferencefrom surrounding light sources.

Lastly, with the resurgence in tablet type computers having pressuresensitive screens, Microsoft® has released an on-screen keyboard intheir latest version of Windows® that enables a user to utilize theirfingers or a stylus to input data.

While each of these systems are a novel approach to overcoming thedependence on a physical keyboard, there are still shortcomings. Namely,each of the systems require the user to either be physically tethered toa computer, where pressure sensitive devices that must be depressed on asurface or find a smooth surface to set up a virtual keyboard.Additionally, these devices do not allow a user to customize the layoutof the keyboard or adapt the keyboard to a users specific style oftyping. In addition to being tethered to a computer, the use of physicalinput devices may cause injury to the user. For example, many claims arefiled every year for repetitive stress injuries incurred from keyboardusage. Examples of common injuries are carpal tunnel syndrome, eyefatigue, neck and back strain, many of which are attributed to usage ofa personal computer.

Attempts have been made to eliminate the use of a keyboard as an inputdevice entirely. Many manufactures have attempted to produce voicerecognition software systems, wherein a user could speak every commandto a computer thereby eliminating the need for a physical or virtualkeyboard. While this approach may be novel, presently voice recognitionsoftware has not advanced to the point of being reliable enough toreplace a keyboard. In addition to requiring more hardware, amicrophone, the voice recognition software is always running within acomputer's operating system, thus requiring additional computing power.Also, voice recognition software must be custom tailored to each user'svoice, inflections and/or accents, therefore once a system has beencustomized to an individual user other user's cannot readily utilize thesystem. Another shortcoming of voice recognition systems is that it isdifficult to use voice recognition for editing, browsing the Internet,graphic design and similar input intensive programs. Additionally,constant talking may fatigue the user's voice, wherein the user's pitchand tone may change, thereby leading to additional input errors becausethe voice recognition software no longer recognizes the user's voice.Further still, voice recognition systems cannot be utilized in cubicletype work environments or similar “open” type environment where noiseinterference from other voices may confuse the voice recognitionsoftware.

Additional input devices may also be utilized in conjunction withkeyboards. For example, pointing devices, such as “mouse” pointingdevices and so called “track ball” devices are also popular computerinterfaces. Generally, these types of devices provide velocityinformation, in both an X direction and an orthogonal Y direction, tothe computer, as well as signals from one or more momentary contact pushbuttons. A pointing icon or other “tool” on a computer monitor respondsto such velocity input by corresponding X and Y movement on the computermonitor. Graphics tablets are another type of “pointing” input devicethat provide the computer with X and Y positional information, asopposed to velocity information, which is used in much the same mannerby the computer. Such devices are well suited for pointing to varioussoftware “push button” options on the screen, selecting portions of textor a group of software “objects,” freehand on-screen drawing,positioning a typing cursor location, and similar functions. However,such pointing devices are remarkably ill suited for text data input.

Other types of computer interfaces have been developed to overcome someof the above-mentioned drawbacks. For example, U.S. Pat. No. 5,212,372to Quick et al. on May 18, 1993, teaches a glove device that has sensorsfor measuring the curvature of each finger at joints thereof. Forentering numerical data, a person using this type of device curves hisfingers to point to “zones,” or virtual keys, that each represents aparticular number. While the input of alphabetical data is mentioned inthe Quick disclosure, only numerical zones are illustrated and itremains unclear how such a device could possibly be used to enter thetwenty-six additional characters of the alphabet, especially since thelittle finger is used solely for designating an “enter” key and istherefore not available for pointing to alphanumeric zones.

A variety of similar glove-based prior art devices exist, and in mostcases each uses some type of joint flexing sensor to determine fingercurvature. Many such devices are designed for communication with deaf orotherwise challenged individuals, and typically provide for computerinterpretation of alphanumeric data formed by a single hand withstandard sign language. It is a slow and fatiguing process for people,even those fluent in sign language, to use such devices to enter a largeamount of data into a computer, such as might be required while typing apatent disclosure, for example. Further, while finger curvature isrelatively easy to detect in a variety of sophisticated ways, suchdetection is only accomplished in one dimension. Lateral movement of thefinger, for example from the “J” key to the “H” key of a standard QWERTYkeyboard, cannot be detected by such joint flexure sensors as disclosedin the prior art. This drawback is also evident in many “virtualreality” data manipulation gloves, which also include a variety ofmotion sensors on similar gloves. As a result, such devices have limiteduse and are not well suited for prolonged data entry from a wideselection of character and command keys, such as those found on thestandard computer keyboard. As previously described, these gloves aregenerally fragile and are not constructed for constant everyday usage.Additionally, the gloves are particularly sensitive to moisture such assweat from the users hands or a wet environment, wherein moisture maycause sensor problems or lead to eventual failure of the glove.

Therefore there is a need for a device that eliminates the shortcomingsof the presently available input devices, wherein the device may beutilized by a user in any physical configuration without requiring theuser to remain physically limited by the device. Such a needed devicewould be adaptable to any individual, regardless of hand size or typingstyle. Further, such a needed device could be used equally well for bothalphanumeric data entry, command entry, and position/velocity input.Such a needed device would be to a large extent softwarere-configurable, making use of the device immensely flexible andadaptable. The present invention fulfills these needs and providesfurther related advantages.

These and other objects, advantages, and features of the invention willbecome apparent to those persons skilled in the art upon reading thedetails of the methods and systems of the present invention, which aremore fully described below.

SUMMARY OF THE INVENTION

The present invention provides systems and methods for interactionwithin a virtual environment. Wherein the methods of the presentinvention may be utilized to virtually interact with a computer systemthrough the use of virtual input devices, wherein the methods andsystems allow a user to input data into a computing system withoutphysical limitations.

In accordance with the present invention there is provided a method forcontrolling a microprocessor based system in a virtual environment, themethod comprising: loading a computer program into a memory space;transmitting at least one signal from at least one transmitter;displaying virtual input devices on a display device, wherein thevirtual input devices initially have pre-determined coordinates;receiving data generated from movement of at least one sensor;calculating coordinates of the sensor movement; comparing calculatedcoordinates to the pre-determined coordinates of each virtual inputdevice; calculating desired input from coordinates generated by thesensor movement; and displaying input on the display device andtransmitting the input to the operating system.

In accordance with another embodiment of the present invention there isprovided a method for controlling a microprocessor controlled device ina virtual environment, the method comprising: loading a computer programinto a memory space; loading an operating system into a memory space;transmitting at least one signal from a transmitting device displaying avirtual keyboard and a virtual input device on a display device;initializing coordinates defining individual keys of the keyboard;initializing coordinates defining a location of the input device inrelation to the keyboard; receiving data at least one sensor wherein thedata received is converted into coordinate information and stored in amemory space; determining if coordinated derived from movementcorrelates to a key location of the virtual keyboard or movement of thevirtual input device; and displaying sensor movement on the displaydevice.

In accordance with the present invention there is provided a method ofgenerating and controlling a virtual workstation, the method comprising:initializing an operating system and a controller system in amicroprocessor based computer system; displaying virtual hands, avirtual keyboard, a virtual pointing device and a virtual workstationenvironment on a display device; monitoring movement of sensors disposedon a user's hands for movement; displaying movement of the virtual handsin response to movement of the sensors; and determining if movement ofat least one sensor passes a user defined threshold.

In accordance with the present invention there is provided a method ofgenerating a virtual gaming system, the method comprising: initializingan operating system and a controller system in a microprocessor basedcomputer system; loading a game program into a memory space; displayinga virtual player on a display device; monitoring movement of sensorsdisposed on a user for movement; and displaying movement of the virtualplayer in response to movement of the sensors.

In accordance with the present invention there is provided a system forvirtually controlling a microprocessor based system, the systemcomprising: a microprocessor based computer system having an operatingsystem configured to be run thereon; a display device, at least onetracking device; at least one sensor, wherein said tracking device isconfigured to track movement of said sensor and determine coordinates ofsaid sensor and time components of sensor movement within a pre-definedarea; and a software component, wherein said software is stored in acomputer readable medium, wherein said software is in communication withsaid tracker and said display device, wherein said software determinesvector movement and acceleration of said sensor and displays said sensormovement on said display device.

In accordance with the present invention there is provided a system forimplementing a virtual reality environment, the system comprising: adisplay device associated with a user, the display device beingresponsive to image data for generating and displaying an imagesimulating a physical computer system, including a physical keyboard, aphysical input device, and physical representation of the user's hands,wherein each of the simulated components appear to the user to be inspace independent of actual physical objects; an output device fortransmitting a signal; an input device for generating data in responseto interaction with the signal; a processor connected to the input andoutput device and the display device and operating a virtual environmentmanager program and a multi-dimensional basic input and output programfor generating a virtual keyboard, a virtual input device, and virtualhands, the processor being responsive to data generated from the inputdevice, for generating motion image data corresponding to the inputdevice data; and wherein the display device is responsive to motionimage data for generating a second image simulating physical motion ofat least one virtual component.

In accordance with the present invention there is provided a system forimplementing a virtual reality (VR) computing environment, the systemcomprising: VR display device including at least one display and worn bya user the one display viewable by the user, with the VR display,responsive to first image data, for generating and displaying a first VRimage simulating a physical computer system including a virtual keyboardhaving a plurality of physical keys, a virtual mouse having at least onephysical button, with the first VR image representing the VR keyboardand VR mouse, the VR keyboard and VR mouse having a first appearancecorresponding to the first image data; an input and an output device forgenerating motion-representative data corresponding to motion of auser's body part; and a processor connected to the VR display device andoperating a virtual environment manager (VEM) and multi-dimensionalbasic input/output subsystem(MD-BIOS) program, wherein the VEM andMD-BIOS provide the first image data to the VR display device, theprocessor being responsive to the motion-representative data generatedfrom the input device, for generating motion image data corresponding tothe motion; and wherein the VR display device is responsive to themotion image data for generating a second VR image simulating motioncorresponding to the motion of the portion of the body of the user.

In accordance there is a need for smaller input/output interfaces asminiaturized portable computing devices become more common.

There is also a need for a system that recreates a full desktopcomputing experience without requiring the space needed for a fulldesktop computer or physically limiting a user to a physical location toutilize such a system.

There is an additional need for a system that can be utilized tominimize computer related injuries such as repetitive stress injury,carpal tunnel injuries and other such injuries that are related tophysical computer use.

There is also a need for a system that is capable of displaying truethree dimensional real world human-machine interactions.

It is the applicant's belief that the present invention addresses theseneeds with novel software and hardware solutions as described in detailbelow.

BRIEF DESCRIPTION OF THE FIGURES

To facilitate understanding, the same reference numerals have been used(where practical) to designate similar elements that are common to theFigures. Some such numbering has, however, been omitted for the sake ofdrawing clarity.

FIG. 1 is a block diagram illustrating exemplary mechanical devices thatmay be utilized with the methods in accordance with the presentinvention.

FIG. 2 is an exemplary embodiment of a computer system in accordancewith the methods of the present invention.

FIG. 3 is an exemplary block diagram of the tracker system in accordancewith the present invention.

FIG. 4 is an exemplary embodiment of the system in accordance with thepresent invention.

FIG. 5 is an exemplary embodiment illustrating an alternative embodimentof a display device in accordance with the present invention.

FIG. 6 an exemplary embodiment of an alternative display device inaccordance with the present invention.

FIG. 7 an exemplary embodiment of a user's hand illustrating thewireless communication of the tracker system.

FIG. 8 is a functional flow diagram illustrating the method inaccordance with the present invention.

FIG. 9 is a functional flow diagram illustrating the interaction betweenthe software and hardware components of the present invention inaccordance with the methods of the present invention.

FIG. 10 is a block diagram illustrating an exemplary embodiment of thememory structure of the multi-dimensional basic input/output system inaccordance with the present invention.

FIG. 11 is a block diagram illustrating the threshold values andcalculation methods of the virtual workstation manager in accordancewith the present invention.

FIG. 12 is a functional flow diagram illustrating the method steps inaccordance with the present invention.

FIG. 13 is an exemplary embodiment of a virtual workstation environmentas seen from a user's perspective, wherein a virtual keyboard is showndisposed three dimensionally over a virtual mouse.

FIG. 14 is an exemplary embodiment of a virtual workstation environmentas seen from a user's perspective, wherein the virtual mouse is showndisposed three dimensionally over the virtual keyboard.

FIG. 15 is a functional flow chart illustrating hardware components of avirtual car control center.

FIG. 16 is an exemplary embodiment of the present invention wherein thevirtual environment manager has been configured as a virtual car controlsystem.

FIG. 17 is an exemplary embodiment of the high-level virtual buttons inaccordance with the virtual car control system.

FIG. 18 is an exemplary embodiment illustrating the high-level virtualbuttons and the second level of virtual buttons in accordance with thevirtual car control system.

FIG. 19 is an exemplary embodiment illustrating the high-level virtualbuttons, the second level, and third level of virtual buttons inaccordance with the virtual car control system.

FIG. 20 illustrates exemplary embodiments of additional second levelvirtual buttons after selection of the high-level virtual button.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before the present invention is described in such detail, it is to beunderstood that this invention is not limited to particular variationsset forth herein as various changes or modifications may be made to theinvention described and equivalents may be substituted without departingfrom the true spirit and scope of the invention. In addition, manymodifications may be made to adapt a particular situation, material,composition of matter, process, process act(s) or step(s) to theobjective(s), spirit or scope of the present invention. All suchmodifications are intended to be within the scope of the claims madeherein.

Methods recited herein may be carried out in any order of the recitedevents that are logically possible, as well as the recited order ofevents. Furthermore, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range is encompassed within the invention. Also, it iscontemplated that any optional feature of the inventive variationsdescribed may be set forth and claimed independently, or in combinationwith any one or more of the features described herein.

All existing subject matter mentioned herein (e.g., publications,patents, patent applications and hardware) is incorporated by referenceherein in its entirety except insofar as the subject matter may conflictwith that of the present invention (in which case what is present hereinshall prevail). The referenced items are provided solely for theirdisclosure prior to the filing date of the present application. Nothingherein is to be construed as an admission that the present invention isnot entitled to antedate such material by virtue of prior invention.

Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin the appended claims, the singular forms “a,” “and,” “said” and “the”include plural referents unless the context clearly dictates otherwise.It is further noted that the claims may be drafted to exclude anyoptional element. As such, this statement is intended to serve asantecedent basis for use of such exclusive terminology as “solely,”“only” and the like in connection with the recitation of claim elements,or use of a “negative” limitation. Last, it is to be appreciated thatunless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs.

Referring to the detail drawings and the disclosure herein, the term“keyboard” is defined herein to include alphanumeric keyboards, subsetsof alphanumeric keyboards, keypads including numerical keypads,telephone and DTMF keypads, security access input devices using buttonsand labels, etc., and so it not limited to QWERTY alphanumerickeyboards. Accordingly, it is understood that the use of the term“keyboard” and the depiction in any of the figures of a keyboard such asa QWERTY alphanumeric keyboard typically used with personal computersand the like is only an example of a keyboard for use, interaction, andoperation by a user for any application of keyboards for input and/oroutput devices. As defined herein, the term “keyboard” is more than aplurality of keys, since a keyboard includes a layout of the pluralityof keys as well as keys, with the layout typically being predetermined.The keys may be associated with symbols such as alphabetical, numerical,mathematical, or other representations, and the keys may includeassociated pictorial or symbolic representations thereupon. Accordingly,a keyboard is not identical to a set of buttons but may be a pluralityof buttons having a layout and a set of symbols associated with each keyor button.

The term “virtual reality” and its abbreviation “VR” are herein definedto include, but not limited to, visual and/or other sensory applicationsimplemented using software and/or hardware to simulate and/or providerepresentations of environments which may be different from the physicalenvironment of the user. Such VR may provide visual and/or multimediazones, worlds, and work areas in which the user and/or other softwareapplications may change and interact representations of elements in theVR environment. For example, in a VR world, a graphical representationof a switch may be changed to represent the flicking or switching of theswitch, which may have an associated switch-flicking sound, which isactivated by flicking the switch. In addition, the VR switching of theVR switch may cause the actuation of other events, either in the VRworld or in actual physical devices and structures; for example, theflicking of the VR switch may cause an actual computer to be turned onor off. Accordingly, the term “virtual reality” is not limited tosimulations or representations of VR devices and information in VRworlds, but may also be extended to physical devices as well as, inhybrid implementations, to both physical and VR devices.

In accordance with the present invention, the detail description of thepresent invention will be divided into sections, wherein each sectionwill be utilized to described components of the present invention. Itshall be understood that the examples described herein should not beconsidered limiting in any manner and should be considered exemplary.

In accordance with the present invention there is provided devices,systems and methods for providing a human interface device that may beutilized to simulate a virtual environment. The human interface deviceincludes hardware components consisting of a tracker assembly, aprocessing unit, and a display device. The software/firmware componentscomprise a virtual environment manager (VEM) and a Multi-DimensionalBasic Input/Output subsystem (MD-BIOS), wherein the software andhardware components are utilized in combination to provide a virtualenvironment. Examples of applications of the present invention include avirtual workstation environment (VWE) wherein a physical computingsystem is simulated in a virtual environment including virtual hands andvirtual input devices such as a keyboard, pointing device or othersimilar input devices. Another contemplated application for the presentinvention is a virtual gaming system (VGS) wherein the present inventioncould be utilized to simulate a gaming environment wherein the usercould virtually interact within the game. The above examples are to beconsidered to be merely exemplary in that the present invention may beutilized in many other applications such as military use, flightsimulation/training, corporate meetings, etc.

The present invention will now be described in greater detail below withregard to the system's individual components.

Hardware

In accordance with the methods of the present invention there will bedisclosed hardware devices that may be utilized in accordance with themethods of the present invention. It shall be understood that many ofthe hardware components are described in a general sense and that manyother types/styles of similar hardware devices may be substituted forthose described herein. For example, as described in the presentinvention a computer system may be embodied as an Intel® centralprocessor unit (CPU) based system a RISC based processor system, thoughit shall be understood that this should not be considered limiting inany manner in that a computer system for use with the present inventionmay be based on similar microprocessor devices.

Referring now to FIG. 1, there is shown an exemplary embodiment ofhardware components in accordance with the present invention. As shownin FIG. 1, the hardware components comprise a microprocessor basedcomputer system, a display device, and a tracking system. In addition tothe hardware shown, the present invention further includes softwarerunning stored in a computer readable medium, wherein the software is inassociation with the hardware components of the system. The software ofcomponent of the present invention will be described in greater detailbelow in reference to the methods of the present invention.

As shown in FIG. 2, the microprocessor based computer system 10 includesa central processing unit 20, memory 30, a communication bus 40, acomputer readable storage device 50 such as an optical storage device,magnetic storage device, flash memory storage device or similar computerreadable storage mediums, and at least one communication port 60. Thecommunication port 60 comprise any one of the following types ofcommunication ports as well as a combination thereof: universal serialbus (USB), IEEE 1394 (firewire), serial port, parallel port, infraredport, 802.11b, 802.11a, 802.11g, Bluetooth® or similar communicationports and devices.

Referring now to FIG. 3, there is shown an exemplary embodiment of thetracker system 80 in accordance with the present invention. The trackersystem 80 comprises a system electronic unit (SEU) 82, at least onetransmitter and at least one sensor each in communication with the SEU82.

The SEU 82 comprises communication ports that are in communication withat least one sensor 86 and at least one transmitter 84. Thecommunication ports may comprise serial, parallel, universal serial bus,firewire® or other similar wired communication ports. The SEU 82additionally includes an analog section and a digital signal processingsection controlled by a computer program stored within an erasableprogrammable memory device. The functionality of the SEU 82 and theinteraction between the transmitter and the sensors will be described ingreater detail in the methods section of the present invention.

As shown in FIG. 3, the SEU 82 is in communication with the transmitter84 and at least one sensor 86. In one embodiment the sensor 86 isconfigured to be coupled to the SEU 82 through the use of a cablecommunication device such as a serial port, parallel port, universalserial bus, firewire port, or similar wired communication ports. Thetransmitter may be coupled to the SEU through similar communicationdevices such as those described above with regard to the sensors.

The transmitter 84 is configured to emit an electromagnetic signal,wherein the SEU 82 controls the transmission rate of the transmitter 84.Transmitters 84 that may be utilized with the current invention arethose shown and described in U.S. Pat. No. 4,742,356, the entirety ofwhich is herein incorporated by reference. The transmitter 84 includes aplurality of radiating antennas for radiating electromagnetic energy.Each of the radiating antennas having independent components fordefining a source reference coordinate frame. The transmittedelectromagnetic field will generally have a transmission range of about0 to 30 feet, more preferably 0 to 15 feet and most preferred about 0 to3 feet. The range of the transmitted magnetic field may be adjustedmanually or automatically in accordance with the methods disclosedherein. It is further contemplated that the transmitter 84 may beconfigured to include more than one transmitting device, wherein the twotransmitting devices would be of different types. For example, thetransmitter 84 may include a first magnetic transmitting device and asecond transmitting device or third transmitting device. The second orthird transmitting devices may be configured to transmit acoustical,optical, or electromagnetic signals. As will be described in greaterdetail in the methods section, the system in accordance with the presentinvention may be configured to automatically choose between the twotransmitters or the system may be manually configured. For example, ifthe user were to utilize the system in an environment having a largeamount of stray magnetic fields that may interfere with the magnetictracker, the system in accordance with the present invention mayautomatically switch to one of the other transmitting devices.

The transmitter 84 is configured to transmit at least one signal. Thetransmitted signal may be electromagnetic, optical, acoustical, inertialetc. in a generally defined field of view. For example, the transmittingdevice may be configured to transmit a signal in a field of view havinga spherical radius of between about 0 to 360 degrees, more preferablybetween about 0 and 270 degrees and most preferred between about 0 and180 degrees. It shall be understood that although the present inventionhas been described as including only a single transmitting device shallnot be considered limiting in any manner and that it is contemplatedthat additional transmitting devices may be utilized with the presentinvention to further expand the field of view of the transmitting deviceand or to increase accuracy and functionality.

Referring now to FIG. 4, there is shown an exemplary embodiment of thehardware components of the present invention in use. As shown in FIG. 4the computer system 10, display device 75, and the tracker system 80,wherein the tracker system 80 comprises the transmitter 84, SEU 82 andsensors 86. In a preferred embodiment the computer 10 is configured tobe embodied in the form of a wearable computer system. Examples ofpreferred computer systems would be based on the Pocket PC® platformdeveloped by the Microsoft® corporation. Although a Pocket PC platformis described as a preferred embodiment this shall not be consideredlimiting in any manner. It is contemplated that a user may be tetheredto a conventional desktop computer system either wired or wirelessly orutilize other computer systems such as an Intel® or AMD® poweredcomputing system, PalmPilot® or other similar computing devices.

Referring now to FIG. 4, there is shown one embodiment of an displaydevice 75 in accordance with the present invention. As shown in FIG. 4,the display device 75 may be configured to include the transmitter 84 ofthe tracker system 80. In one embodiment the transmitter 84 isconfigured to be retained on the display device 75. The display device75 in a preferred embodiment is a liquid crystal display (LCD) device,wherein the LCD device is in communication with the computer system 10through the use of a wired or wireless connection (not shown). Inalternative embodiments, the display device 75 may comprise other typesand styles of head mounted display devices, such as organic displays,thin film transistor (TFT) displays, light emitting diode (LED)displays. Additionally, it is contemplated that the display mayincorporate a multi-layer device, wherein one layer is generally opaqueand a second layer is a generally opaque dot-matrix layer, wherein theopaqueness of the first layer may be manually or automatically adjustedthereby allowing the heads-up display to be utilized in many differentambient light situations. While the present invention has been describedas utilizing a single display device, it is contemplated that a seconddisplay device may be utilized for stereoscopic vision. The seconddisplay device may be a conventional display device such as a cathoderay tube (CRT) device, a liquid crystal display (LCD) device, or a videoprojection device, wherein the transmitter 84 would be mounted onto thedisplay device.

It is further contemplated that, a multi-layer liquid crystal displaymay be utilized in accordance with the present invention, wherein themultiple layers are capable of simulating three-dimensions. By utilizingtwo display devices a three dimensional workspace may be simulated. Sucha system could be utilized to simulate a “real” world experience in thevirtual environment and would further provide haptic feedback. As such,the three-dimensional display device would be configured to interactwith the software components of the present invention. The presentinvention would utilize a display device having two LCD screens, whereina left view model is generated and a right view model is generated by agraphic processing unit (GPU). An example of such a system can be seenin FIG. 5, where there is shown an exemplary embodiment of the threedimensional system in accordance with the present invention. As shown inFIG. 5, a user's right and left eyes would focus on the right and leftview model generated in the display device, wherein the user's brain cancomprehend the information displayed on the display device and decidethe distance and depth of the displayed object. Thus, the user wouldbelieve that they are seeing a true three dimensional display of theobject displayed on the display device.

Referring now to FIG. 6, there is shown an alternative display device inaccordance with the present invention. As shown in FIG. 6, the displaydevice may be a conventional display device such as a cathode ray tube(CRT) monitor or a liquid crystal display (LCD) monitor, wherein thetracker would be in association with the monitor frame. For example, thetransmitter maybe positioned in the upper corner of the monitor frame,wherein the physical location of the transmitter will be utilized toestablish an origin for the coordinate system of the present inventionas will be described in greater detail. Further still, it iscontemplated that the transmitter may be placed in other locations onthe frame of the display device or alternatively in a location near thedisplay device. In addition to conventional display devices as describedabove, other types of display devices may be utilized. For example, avideo projector may be utilized to project an image on a surface,wherein the transmitter will be placed near one corner of the projectedimage. The projected image may be directed onto a screen, wall orsimilar vertical surface, or alternatively, the projector may be mountedsuch that the image is projected onto a horizontal surface such as aconference table. Further still, the display device may be a combinationof devices, for example, a video camera and a CRT or LCD or projector,wherein the present system and methods may be utilized for videoconferencing.

In another embodiment, it is contemplated that the display device may beembodied in the form of a physical display device such as a white board,chalkboard or a similar device, wherein the transmitter would be mountedto one corner of the board.

Still further, it is contemplated that any of the display devicesdescribed above may be utilized in any combination. For example, a usermay utilize a head mounted display device and a projector simultaneouslyin accordance with the methods described in the present application.

As shown in FIGS. 1-6 the present invention further includes at leastone sensor device 86 disposed on the user. The sensor device 86 ispreferably disposed on the user's hand, and most preferable disposed onthe user's fingertip. It is contemplated that at least one sensor 86 maybe disposed on each of the user's fingertips. As shown and described inFIGS. 3 and 4, the sensors are in communication with the SEU 80.

The sensor(s) 86 are comprised of multiple coils encased in a protectivehousing, wherein the coils are configured to interact with theelectromagnetic field generated by the transmitter 84. Each of thesensors are configured to generate at least one electrical signal, andmore preferably between about three and six electrical signals inresponse to interaction with the signal generated by the trackingdevice. The electrical signal(s) generated by the sensor 86 are passedthrough the cable connection to the SEU 82, wherein the signals areamplified and converted into a digital signal. In addition to convertingthe sensor signals into digital form, the MD-BIOS further assigns aunique sensor id to each sensor. The digital signals are embodied in theform of coordinate information of the sensor, such as, x, y, z, and yaw,pitch and roll information. This coordinate information is then passedfrom the SEU 82 to the computing system 10.

Referring now to FIG. 7, there is shown an alternative embodiment of thetracker system 80 in accordance with the present invention. Wherein inthe alternative embodiment it is contemplated that the sensors 86 may beconnected to a transducer 81, wherein the transducer is configured toconvert the analog signals to digital signals and wirelesslycommunicates the digital data to the SEU 82. In one embodiment inaccordance with the present invention, the transducer 81 is configuredto be hard wired to at least five sensors. In this configuration thetransducer would assign a unique sensor identification (ID) to eachsensor, wherein this unique sensor ID would be transmitted to the SEU 82along with the coordinate data of each sensor. Although the transducerhas been described as being wirelessly coupled to the SEU 82 it iscontemplated that it may communicate with the SEU 82 through a wiredcommunication port. It shall be understood that more than one transducer81 may be utilized with the system in accordance with the presentinvention. For example, sensors may be disposed on each fingertip of auser's hands, wherein at least two transducers 81 will be utilized, onetransducer 81 being disposed on each of the user's hands and inassociation with the sensors disposed on each of the user's fingertipsrespectively.

Although the SEU 82 has bee described as being an independent componentseparate from the computing system 10 it is contemplated that the SEUmaybe integrally formed with the computing system. For example, the SEUmay be configured to be a removable communications card in the form of aPCMCIA card, compact flash card, PCI card or other similar removabledevices. Alternatively, SEU may be integrated into the system board ofthe computing system.

Although specific hardware has been described in conjunction with thepresent invention, it shall be understood that it is contemplated thatvariances in the described hardware may be undertaken without departingfrom the scope of the present invention.

Methods

In accordance with the present invention, methods of use of the presentinvention will be described in detail below, wherein it shall beunderstood that these components will be described in a general senseand should not be considered limiting in any manner. In accordance withthe present invention, various mechanical components are utilized inconjunction with software and electronic components to define the systemand methods in accordance with the present invention.

In accordance with the present invention there are provided softwarecomponents configured to control the various hardware component of thepresent invention wherein the software and hardware components togetherform the system of the present invention. Referring now to FIG. 8 thereis shown an exemplary functional flow chart illustrating the interactionbetween the various software components of the present invention. Asshown in FIG. 8, the software components in accordance with the presentinvention comprise a virtual environment manager (VEM) 100 and amulti-dimensional basic input/output system (MD-BIOS) 110, wherein thesoftware components are configured to interact with the hardwarecomponents described above. Furthermore, it is contemplated that thesoftware components of the present invention will be embodied in acomputer readable media such as a cd-rom, dvd, hard drive, flash memory,programmable read only memory or any other type of computer readablemedia.

As shown in FIG. 8, the VEM 100 receives coordinate data and a timestamp from MD-BIOS 110, wherein the VEM 100 utilizes the coordinate datato simulate virtual device actions and display the virtual deviceactions on the display device. Additionally, MD-BIOS 110 and VEM 100 arein communication with an operating system. It shall be understood thatthe VEM 100 can be tailored for each application purpose. For example,if the system in accordance with the present invention is configured tovirtually simulate a computer workstation, then the VEM 100 would beconfigured as such to generate a virtual workstation. Additionally, if agaming system is to be replicated then the VEM 100 would be a virtualgaming manager, is a conferencing system is to be replicated than theVEM would be configured to be a Virtual Conferencing Manager. It shallbe understood that the above examples should not be considered limitingin any manner, in that they have been provided for exemplary purposesonly.

Referring now to FIG. 9, there is shown an expanded embodiment ofMD-BIOS 110 in accordance with the present invention. As shown in FIG.9, MD-BIOS 110 receives coordinate data from the SEU, wherein MD-BIOSadds time stamp information to the coordinate information. Referring nowto FIG. 10, there is shown an exemplary embodiment of the memorystructure of MD-BIOS, wherein data received from each sensor isinterpreted and placed into appropriate memory location in the memorystructure as shown. MD-BIOS is capable of receiving data from a multiplenumber of sensors, wherein each sensor is assigned a sensoridentification tag (id) by the MD-BIOS or transducer depending upon thesystem's configuration.

Applications that utilize MD-BIOS not only can read the sensor id's andtheir properties, but can also check the sequence number to know thedifference between each input to the application. Applications can alsointeract with MD-BIOS in one of two ways or in a combination of each.For example, it is contemplated that multiple programs running in amemory space of the personal computer may utilize different sets ofsensors. For example, in association with the virtual workstationembodiment, a number of sensors associated with the user's fingertipsmay be utilized to control the virtual keyboard and mouse in a wordprocessing program, while another program may be collecting data fromother sensors associated with the present invention. For example, thepresent invention may be utilized in a laboratory setting, wherein inaddition to manually entering data by utilizing the virtual keyboard, ascientist may wish to automatically collect data from an experiment orroom conditions. In this embodiment, the scientist would disposeadditional sensors to measure to desired properties. For example, onesensor would be utilized to measure room temperature, another forhumidity, another to measure ph or another chemical property of anexperiment, etc.

The first method of program interaction is referred to as synchronizedinteraction. In the synchronized method, if one or more applications areinterested in the same set of sensor id's the applications need toregister to MD-BIOS to listen to the set of sensor id's which eachapplication has interest and leave a call-back function address withMD-BIOS. Therefore, whenever data is updated, MD-BIOS interrupts theapplication to acquire the data, then the application resumes processingthe data. This provides synchronous operation of data processing anddata acquisition. In the asynchronized method, if one or more unequalfrequency applications are interested in the same set of sensor id's,MD-BIOS will filter out the application that requires a higher samplingfrequency. MD-BIOS will carry out its best resolution, if the requestedfrequency is outside the resolution of the tracker, then MD-BIOS willeither return an error or will return a reduced frequency rate. In thiscase, some of the lower frequency applications may need to know the dataacquisition gap and therefore utilize the sequence number of the sensorid' to determine the frequency.

Additionally as previously discussed and shown in FIG. 5, MD-BIOS mayadditionally be utilized to create a true three dimensional virtualenvironment through the use of at least two display devices, whereinMD-BIOS generates a left and right view which are then displayed on acustomized display device as previously described in the hardwaresection above.

Further still, it is contemplated that MD-BIOS may be further configuredto control the transmitter in combination with the system electronicunit, wherein the two systems could be utilized to adjust thetransmission frequency of the transmitter or switch between the varioustransmitter types. As described herein in accordance with the presentinvention, the transmitter is configured to emit an electromagneticsignal that the sensors interact with and produce an electrical signalthat is converted into the coordinated received by MD-BIOS. MD-BIOScompares the coordinate information against the previously receivedcoordinate information for each sensor, if the deviation of each sensoris greater than a predetermined amount, for example, sensor movement ofa quarter of an inch in one computing cycle at 120 Hz would beconsidered to be excessive. Therefore, MD-BIOS would automaticallydirect the transmitter to energize the alternative transmitting devices,wherein MD-BIOS would then sample the coordinate generated by alltransmitting devices, if one set of coordinate information is outsidethe parameters as described above then that coordinate information isdiscarded and the other two sets of coordinate information are compared.If the two remaining coordinate information sets are similar, MD-BIOSchooses one of the transmitting systems and turns the remaining systemsoff. Therefore, MD-BIOS will automatically switch between the individualtransmitting devices if interference is detected. Typically, MD-BIOSwill automatically switch without the user's knowledge, though it iscontemplated that the user may specify a desired system and adjust thedefault values to their preferences.

In accordance with the present invention the Virtual Environment Manageris utilized to generate a virtual workstation environment. The virtualworkstation includes a virtual keyboard at least one virtual inputdevice. As described above MD-BIOS receives coordinate data from thesensors adds a time stamp to the coordinate information and stores thecoordinate data in a memory structure as shown in FIG. 10. The virtualenvironment manager includes a plurality of action thresholds todetermine if sensor movement is to be interpreted to be an intended userinput or is sensor movement can be attributed to inherent user movementsuch as natural heartbeat or muscle twitching. Initially, thesethreshold values are set at default levels, though these default levelsmay be adjusted by the user to suit the user's own preferences.

Referring now to FIG. 11 there is shown threshold values that the VEMutilizes for determining if sensor movement correlates to intended userinput. After receiving coordinate data from MD-BIOS, the coordinate datais utilized to determine whether the sensor movement is to be returnedas a key press or other intended input or if the sensor movement is dueto natural movement. Initially, VEM generates a virtual keyboard havinga known origin, wherein the coordinates generated for each of thesensors are compared to the origin coordinates to determine where theuser's hands are in relation to the keys of the virtual keyboard. Avirtual input device, such as a mouse is also generated wherein a set oforigin coordinates are also established for the virtual input device,thereby allowing VEM to determine sensor location in relation to thevirtual keyboard and virtual input device.

As shown in FIG. 11, the key press threshold is a vector propertywherein a value of less than negative one millimeter of a sensor'scoordinates will be interpreted as a key press, wherein VEM will comparethe sensor's coordinates to the origin coordinates of the virtualkeyboard or mouse to determine the proper key press. The key press willthen be transmitted to the appropriate program, such as a wordprocessing program, text-editing program, graphics program, etc. . . Akey release will be interpreted as a vector having a value of greaterthan 1 millimeter and having duration greater than one second. A keyhold will be determined by MD-BIOS as a vector having a value equal toone millimeter.

Movement of the virtual input device is determined by vector movement ofthe sensors, if the coordinate values of the sensors are moved less thanone millimeter then VEM will not consider this to be a mouse movement,if the coordinate movement is greater than one millimeter this will beinterpreted as a mouse move. To determine if the user intends to depressa button on the mouse, VEM utilizes the threshold values which have beenpre-established for key presses on the virtual keyboard.

In accordance with the present invention a series of functional flowdiagrams will be described herein, wherein the functional flow diagramsare utilized to illustrate the interaction between the variouscomponents of the present invention.

Methods

In accordance with the present invention there is provided methods for avirtual computing environment. Wherein a virtual environment managerwill be embodied in the form of a virtual workstation manager. That isthe present invention will be utilized to replicate a personal computingsystem and the physical input devices in a virtual environment. As willbe described in detail below, the virtual workstation manager willgenerate virtual input devices such as a keyboard, a mouse or otherinput devices.

Referring now to FIG. 12, there is shown an exemplary embodiment of amethod in accordance with the present invention, wherein the VEM andMD-BIOS are embodied in a computer readable medium. The method accordingto the present invention comprises the steps of: initializing a computersystem, including initializing the SEU, loading an operating system intoa memory space, loading VEM and MD-BIOS into a memory space. As soon asMD-BIOS is loaded into a memory space, MD-BIOS begins to receivecoordinate information from the SEU; VEM then begins to scan for sensormovement to determine if any sensor movement passes the thresholdvalues. A display device is initialized, wherein VEM (such as VWM) thendisplays virtual hands, virtual keyboard and at least one virtual inputdevice on the display device. MD-BIOS each of these processes will bedescribed in greater detail with reference to detailed functional flowdiagrams.

The methods according to the present invention may utilize hardwaredevices such as those described above or hardware devices similar tothose shown and described above, wherein the present invention utilizessoftware or firmware to control the hardware in a manner to replicate avirtual environment. In accordance with the present invention comprisesat least one software program or firmware code previously describedherein wherein the software includes the VEM and MD-BIOS programs

Referring now to FIG. 12 there is shown a functional flow diagramillustrating exemplary steps of the method in accordance with thepresent invention. At Box 190 a computing system is powered on, thisincludes powering on the SEU additionally if the SEU is embodied as anindividual component separate from the computer system. As Box 200 anoperating system is loaded into a memory space. At Box 210 MD-BIOS andthe Virtual Environment Manager are loaded into a memory space withinthe computer system. After loading MD-BIOS and the virtual workstationmanager into a memory space, MD-BIOS immediately begins receivecoordinate information from the SEU at Box 215. As described in detailabove the electrical signals generated by the sensors are converted tocoordinate information and a time stamp is added by MD-BIOS and the datais stored in a memory location. VEM then compares the coordinates ofeach sensor to determine if motion of the sensor(s) has occurred andwhether the sensor motion is intended to be a key press on the virtualkeyboard or virtual mouse. The transmission and scanning rate of thetracker is controlled by MD-BIOS. Wherein the scanning and transmittingfrequency of the tracker is controlled in response to the physicallocation of the sensor(s) in relation to the tracker and the origincoordinates of the virtual devices. For example, as the sensors aremoved closer to the virtual devices the scanning rate of the trackerwill be increased, thereby increasing the accuracy and resolution of thesystem. When the sensors move away from the virtual devices, thescanning rate is reduced, thereby lowering the power consumption of thedevices. In addition to controlling the scanning rate, VEM also may beconfigured to display a visual indicator of the sensor(s) position onthe display device as well as provide auditory feedback to the user. Forexample, as the sensor(s) are moved to a location above the virtualkeyboard, the sensor may change colors to indicate how close the sensoris to the keyboard, when the sensor moves toward the keyboard, VEMchanges the color of the sensor to indicate a key press and provides anauditory response to the user to denote a key press.

In an alternative embodiment the virtual keyboard may be configured torespond to absolute movement of each of the user's fingers (sensors). Inthis embodiment, the threshold values are set to zero or near zero,wherein any movement of the user's finger will correlate to a typingmotion or mouse motion depending upon where the user's fingers arelocated in relation to the virtual devices.

At Box 230 the system prompts the user to enter the user's login andpassword. At Diamond 235 it is determined If the user already has apassword and login then Box 237, where the user's saved settings areloaded into a memory space. A user's settings may control what isdisplayed on the virtual display device such as input devices (keyboard,mouse, tablet), the location of these input devices in the virtualsetting and any other user definable preferences such as visual or audiofeedback, tactile feedback and sensitivity of sensor movement, or otherpreferences. For example, user A may have saved settings from a previoususe, wherein once logged into the system, VEM (such as VWM) displays avirtual keyboard, a virtual monitor and a virtual mouse, wherein thevirtual workstation manager controls the origin coordinates of whereeach of these devices will be displayed in the virtual environment. Ifit is determined in Diamond 235 that a new user is logging in then atBox 239 default values for the virtual devices displayed on the displaydevice are loaded into memory. At diamond 242, the system prompts theuser whether or not they want to customize the default values. If theuser chooses to customize the default values then Box 243. At Box 243the user's customized settings are saved under their login profile andthen to Box 240. If the user does not choose to customize the defaultsettings then Box 240.

At Box 240 the display device is initialized and at Box 250 the virtualdevices are displayed in the virtual environment at the locationdictated by either the user's settings or the loaded default values.

If the user is a previous user, the virtual environment manager will becalled wherein the user's preferred settings will be retrieved from amemory location and loaded into a memory space. If the user is new tothe system, the Virtual Environment Manager will be continue to usesystem default values into memory. These default values may be changedby the user and saved under the user's personal profile before exitingthe system, wherein these settings will be loaded when the user utilizesthe system in the future.

Examples of user definable settings are the type and style of keyboardthat will be displayed in the virtual environment. For example, the usermay prefer to utilize a standard size keyboard having a standard QWERTYlayout, or the user may prefer to utilize an ergonomic keyboard.Additional settings would be the physical location of the keyboard inthe virtual environment. Such that when the system is initialized thekeyboard will be shown in the virtual environment wherein thecoordinates of the keyboard are known. In addition to displaying akeyboard as a user setting, additional settings may control the displayof other virtual input devices such as a mouse, a virtual monitor orsimilar devices. Each of the virtual devices displayed within thevirtual environment will have origin coordinates known to the virtualenvironment manager. The user in the virtual environment may control thestyle, size and location on the virtual input devices. For example, thekeyboard may include handles, wherein the user can grab the handles inthe virtual environment and move, stretch, pull-apart the keyboard. Theother virtual input devices may also include handles to allow thedevices to be re-positioned re-sized or stylized within the virtualenvironment. When the computer system is powered down, the virtualenvironment manager can saves the user's settings, wherein the virtualenvironment may be loaded each time as left by the user in each previoususe

Once the user has logged into the computer system, the operating systemand virtual environment manager and MD-BIOS finishes loading into memorywith the user's settings. After loading the operating system and MD-BIOSinto memory, the display device is initialize, Box 240, wherein agraphical user interface (GUI) is displayed on the display device Box250. For example, if the operating system is based on Microsoft Windows®then the standard GUI interface will be displayed in the display device.In addition to displaying the operating system's GUI, virtual inputdevices will also be display on the display device. As described above,the placement, size, shape and orientation of the input devices asdisplayed within the virtual environment is dictated by either the userssaved preferences or in the case of a new user the default settings ofthe Virtual Environment Manager. In addition to displaying virtual inputdevices on the display device, a virtual set of hands may also bedisplayed if the user is wearing sensors on each of their fingertips.

Referring now to Box 260, the transmission rate of the signal iscontrolled by MD-BIOS and the virtual workstation manager in conjunctionwith the coordinate location of the sensors in relation to the tracker.The transmission rate of the tracker may be controlled according tosensor position. For example, if it is determined that the sensor isdisposed near the known location of an input device, then the tracker'sscan rate will be increased to increase accuracy of the system, if thesensor is disposed at a given distance from any virtual input devicethan the transmission rate of the tracker will be reduced, therebysaving power and reducing processing cycles. Further still, as describedabove, if coordinates received from the SEU are outside the operatingparameters of the system, MD-BIOS will direct the transmitter toenergize the alternative transmission sources, wherein the coordinatesgenerated from these alternative transmission sources will be comparedand the strongest or more accurate coordinate information will beutilized. Wherein MD-BIOS will direct the transmitter to turn off thealternative transmitting devices. In a preferred embodiment this processwill occur automatically without any user interaction.

Referring now to Box 270, each of the sensors are constantly sampled bythe SEU and coordinate data is sent to MD-BIOS where a timestamp isadded and the information is placed into a memory location.

At Diamond 280, the virtual environment manager and determines if thevelocity component(s) and the vector component(s) of the detected sensormotion pass threshold values. Threshold values are utilized to filterout undesired input as well as increase system accuracy. For examplethere are natural vibrations associated with humans such as heartbeat,slight hand motion or other natural vibrations. If it is determined thatthe vector components are greater than the threshold values, then it isdetermined if the coordinates are in the vicinity of the knowncoordinates of the virtual input devices generated by the virtualenvironment manager. For example, if the coordinates received correlateto a keyboard position for a letter, the velocity component and theprevious coordinates are compared to determine the intentions of theuser. By comparing the previous coordinates and the present coordinatesin addition to the velocity component it can be determine if the motionis intended to replicate a keystroke, key release, key hold or keyrepeating. If the coordinates correspond to the location of the mouse,then the vector and velocity components will be utilized to determine ifthe user intended a mouse move, a mouse button depress/release, or asingle or double click.

The movement of the sensors will be displayed on the display device inthe form of a virtual set of hands or in the case of mouse motion,movement of a pointer within the virtual environment. In addition todisplaying motion within the virtual environment, it may also bedesirable to provide feedback in the form of audio, tactile, or visualfeedback of interaction between the user and the virtual input devices.For example, if a key is determined to be depressed on a keyboard, thenthe virtual keyboard will be displayed wherein the chosen key isillustrated as being depressed. In addition to the visual feedback,audio or tactile feedback maybe provided in the form of “clicking”sounds intended to replicate the physical sound of a standard keyboard.

At Box 300 and 305 the coordinate information generated from sensormotion is compared to the known coordinate information of the virtualinput devices. For example, if a virtual keyboard is displayed accordingto the user's defined settings, coordinates of the keyboard are a knownvalue that are stored within the shared memory manipulated throughMD-BIOS API calls. Thus, when coordinates are calculated from sensormotion, the coordinates transmitted from the memory location of MD-BIOSare compared to the known coordinates of the keyboard. If thecoordinates correspond to or are within a pre-defined range of anindividual key disposed on the keyboard then the virtual environmentmanager will determine the user's intention as indicated in Box 330. Thevirtual environment manager determines the user's intention by comparingthe velocity component generated by sensor movement and comparing thevelocity component to a known value. For example, if the vectorcomponent is toward the plane of the keyboard, then the virtualworkstation manager will interpret this to indicate a key press on thevirtual keyboard. If the vector component is away from the plane of thekeyboard it may also indicate that a key release was performed and theuser's finger is moving away from the key. If the velocity component iszero or nearly zero this may be interpreted to indicate a key hold, suchas a user holding down a shift key or the like. A vector of zero mayalso indicate a key repeating such as when a typed word includes two ormore of the same characters. After the user's intention is determinedthen at Box 305, the intended information is sent to the operatingsystem for entry into the event queue or windows manager. In accordancewith the present invention, the key presses may be determined utilizingpure vector and velocity without consideration of the z-axis component.This would allow a user to utilize the present system while restingtheir hands on a surface, such as a tabletop or upon their person or thelike.

At Box 300, the user's intended motion may be displayed on the virtualdisplay device in the form of a virtual key-press on the virtualkeyboard, motion of a virtual hand or pair of virtual hands, or anauditory or tactile response may be generated in response to thesensor's motion.

At Box, the system returns to Box 270, wherein the system returns toscanning for sensor motion and the process is repeated until the systemis powered off.

It shall be understood that although the sensor motion component hasbeen described herein in reference to a key press on a keyboard it shallbe understood that this should not be considered limiting in any manner.In that the coordinates may correspond to any type of virtual inputdevice, wherein each virtual input device has parameters which areutilized by the virtual environment manager to interpret the sensormotion. For example, if the coordinates received correlate tocoordinated for a mouse, it will be determined if the coordinates definemovement of the mouse, which will be displayed within the virtualenvironment. A mouse button press, mouse button release, a single buttonpress or a double button press.

It is further contemplated that the virtual environment manager may becapable of distinguishing between different sets of sensors by eachsensor(s) or transducer(s) unique ID. For example, if two users bothutilizing a system in accordance with the present invention are near oneanother each tracker will only track those sensors to which the systemis initialized. Thus, multiple systems can be utilized within closeproximity to one another.

As shown in FIG. 12, the functional flow diagram illustrates a graphicbased system, wherein the virtual reality system in accordance with thepresent invention is configured to display three-dimensional virtualinput devices in a virtual environment. It is further contemplated thatthe present invention may be utilized in a text-based system. In a textbased system the functional flow diagram as shown in FIG. 12 would notinclude Box 200 wherein the operating system is loaded into a memoryspace. In the text-based system the virtual keyboard generated by thevirtual environment manager would be displayed one dimensionally and thevirtual input device in a preferred embodiment would be a touch pad or asimilar tracking device.

It is further contemplated that a portion of the software components ofthe present invention may be embodied in the form of a BIOS chip,wherein the BIOS chip would be installed on the computer system's motherboard, such that when the computer system is powered on the software inaccordance with the present invention would be loaded into a memoryspace from the BIOS.

Referring now to FIGS. 13 and 14 there are shown exemplary embodimentsof various virtual keyboards as displayed in the virtual environment. Asshown in FIG. 13, the virtual keyboard 300 may be disposed over the topof the virtual mouse 310 thereby allowing a user to immediately togglebetween the keyboard and the mouse without having to physically move thesensors a great amount. This type of setup reduces motion associatedwith switching back and forth between the keyboard and the mouse,thereby potentially reducing repetitive stress injuries. Further stillby providing such a layout, users of programs that require a user toswitch back and forth between a pointing device and text input may beable to increase their productivity because less time is spent switchingbetween the two devices. Referring now to FIG. 13, there is shown thevirtual keyboard 300 and virtual mouse 310, wherein the virtual mouse isdisposed over the virtual keyboard. As shown in FIGS. 13 and 14 thevirtual mouse can be transposed between the two positions shown eitherthrough specific hand motions wherein the sensors and VEM generate aspecific signal to transpose the mouse from one position to another.Alternatively, the mouse may be transposed utilizing a voice command, ahot key associated with the virtual keyboard or a hot spot located onwithin the virtual display. Additionally, although a conventionalcomputer mouse is shown being displayed in the virtual environment, itis contemplated that any type of input device may be displayed, forexample the input device may be a touch pad, trackball, tablet andstylus or similar input devices.

Referring now to FIGS. 15 and 16 there is shown yet another applicationof the system in accordance with the present invention, wherein thesystem of the present invention is configured as a virtual car controlsystem (VCCS) 400. In the present embodiment, the VCCS comprisessoftware and hardware components, wherein the software and hardwarecomponents interact to provide a virtual system that enables a user tocontrol various systems within an automotive environment withoutphysically removing their hands from the steering wheel and shiftingtheir eyesight from the road.

The VCCS system 400 includes hardware and software elements to replicateand control mechanical and electrical controls within an automotiveenvironment. The VCCS includes a computer unit 410, a tracker system 450including a system electronic unit 420, at least one sensor 425 and atleast one transmitter 430, a control button 460 and a heads up displaydevice 470 or alternatively a see through HMD. The VCCS further includessoftware components 480 stored on a computer readable medium disposedwithin the computer unit 410. The software components include anoperating system 482, a virtual environment manager program 484 and amulti-dimensional basic input/output subsystem (MD-BIOS) 486 program.

Referring now to FIG. 16 there is shown an exemplary embodiment of theVCCS as seen from a user's perspective in accordance with the presentinvention. As shown in FIG. 16, an image showing three-dimensionalbuttons 500 would be projected onto the windshield 505 by a heads updisplay device 470, when the buttons are displayed on the windshieldcoordinate information is associated with each of the buttons. Thecoordinate information of each button locates each button in space in aplane spaced apart from the steering wheel, this can be seen by thethree-dimensional buttons disposed on box 510 which are shown forexemplary purposes only and will not be actually seen by the user. Byplacing the origin coordinates for each button in a plane aligned withthe physical steering wheel of the vehicle, the user may interact withthe buttons without removing their hand's from the steering wheel. Inaddition to the three-dimensional buttons displayed on the windshield,the VCCS may further include at least one physical button. In apreferred embodiment the physical button(s) would be disposed on thesteering wheel or steering column. The physical buttons may be utilizedto turn the heads up display on and off, menu return, enter, clear,dial, send or any similar command which may be utilized to control anyaspect of the VCCS system.

In use, at least one sensor would be disposed on a user's person, in apreferred embodiment at least one sensor would be disposed on each ofthe user's thumbs, but it is contemplated that additional sensors may bedisposed on the user's additional fingers. The sensor(s) interact with asignal generated from a transmitter, wherein the sensors produce aposition signal due to interaction with the transmitted signal. Theposition signal is converted into coordinate information by the SEUafter being transmitted to the SEU. The coordinate information istransmitted to the MD-BIOS software where time stamp data is added andthe coordinate information is stored in a memory location. It is furthercontemplated that a sensor identification tag may also be associatedwith the sensor, added by the SEU or added by MD-BIOS to the coordinatedata. The coordinate data is then utilized by the VEM to determine ifmotion of a sensor passes threshold values, and if so, what is theuser's intent, the coordinate data is then compared to the origincoordinate information for each of the three-dimensional buttonsvirtually located in a plane adjacent to the steering wheel. A virtualhand may also be displayed on the windshield to illustrate the motion ofthe sensor or a cursor or other visual marker may be displayed on thewindshield so that the user may visually correlate their hand locationon the steering wheel with the virtual three-dimensional buttonsdisplayed on the windshield.

As described above the VCCS system in accordance with the presentinvention provides a virtual interface allowing a user to controlvarious components of an automobile. Each of the buttons would bedisplayed on the windshield as having three-dimensional characteristics.Wherein multiple layers of buttons may be utilized to control varioussystems. For example, as one button is pressed, additional buttons maybe displayed on the windshield, wherein the additional buttons aredisplayed as being tiled over the previous layer of buttons. This may bebetter understood as shown in FIGS. 17-19 wherein there is shown anexemplary embodiment of the VCCS system in use in accordance with thepresent invention.

Referring now to FIG. 17 there is shown a series of three-dimensionalbuttons as they would be displayed by the heads-up display device andprojected onto an automobiles windshield. As previously described, thedisplay device or the VCCS system may be controlled by a physical buttonwherein, after the button has been depressed the buttons would bedisplayed on the windshield. The buttons displayed would have origincoordinates that would place the buttons in plane space adjacent to theuser's left hand on the steering wheel wherein the first level ofbuttons displayed would be high-level system buttons. For example, onebutton will be displayed to access the climate control system, anotherfor the GPS system, and another for an entertainment system and so on.It shall be understood that the list above should be consideredexemplary and should not be considered limiting in any manner. If theuser desires to access the entertainment system, they would move theirthumb to a physical location which would correlate to the virtualentertainment system button, wherein motion of their thumb over thevirtual entertainment button would be deemed to be a button press by thesoftware components of the present invention. As a result, a second setof virtual buttons would be displayed by the heads-up display device andprojected onto the windshield as shown in FIG. 18. The additionalbuttons displayed would have origin coordinates that would place thebuttons in plane space adjacent to the user's right hand on the steeringwheel. As shown, the additional buttons would provide the user withaudio/video components as installed, such as AM/FM radio, cassette,compact disc, DVD audio and the like. As shown in FIG. 19, the user hasselected the CD button, wherein the visual display of the CD button ischanged, for example the button is made larger thereby appearing to bein a different plane than the other buttons and the color of the buttonmay change to indicate that the button has been selected. Additionally,it is contemplated that an audible feedback may also be associated withbutton selection in addition to the visual feedback described above.

As shown in FIG. 19, after selecting the CD button, additional buttonswill be displayed adjacent to the selected CD button wherein theadditional buttons include controls associated with the CD player. Forexample, fast forward, skip, rewind, disc selection, mute, volume andmenu return.

Referring now to FIG. 20 there is shown exemplary embodiments ofadditional buttons that may be displayed in accordance with the VCCSsystem of the present invention. As shown in FIG. 20, additional buttonsmay be displayed to control a navigation system such as a GPS system,Climate control system, communication system such as a cellular phone,as well as automotive control systems such as traction control, systeminformation and the like. As shown in FIG. 20, each of the top levelbuttons have been selected thereby causing the second menu to bedisplayed on the screen. It shall be understood that the menus shown inFIGS. 16-20 should be considered exemplary and not limiting in anymanner in that the menus may be altered without deviating from the scopeof the invention.

Although the present application has been described in detail withreference to an entertainment system this shall not be consideredlimiting in any manner. Further still, it is contemplated that the VCCSsystem may be utilized in combination with mechanical switches orphysical control means. For example, the entertainment system may becontrolled with the UCC system by the driver and also may be controlledby the driver or a passenger with conventional means installed in theautomobile. As described previously, the heads-up display may becontrolled by a mechanical switch, a virtual switch or the display maybe time controlled, wherein once activated, if no input is received fromthe user, the display will turn off, wherein the user can thenre-activate the system. It is further contemplated that the system mayhave additional restrictive controls such as a maximum speed control,wherein the system may not be accessed if the vehicle is travelingbeyond a pre-set speed, or if the steering wheel has been rotated past apre-set degree. Additionally, the VCCS system may utilize voice commandsoftware in addition to those hardware and software components describedabove.

Although the coordinate information of the present invention has beendescribed herein as being restricted to the x, y, and z-axis, thesensors are further capable of generating yaw, pitch and roll coordinateinformation in combination with the hardware and software components ofthe present invention. The additional coordinate information may beutilized by applications not mentioned specifically herein. For example,if the virtual environment manager of the present invention was tailoredto represent a virtual flight training simulator, the yaw, pitch androll coordinates would be utilized to simulate movement of a virtualplane.

Although the present invention has been described with reference to avirtual workstation such as a personal computer or laptop computer it iscontemplated that the system and methods in accordance with the presentinvention maybe utilized to replace other types of physical systems.Such as Virtual Game station or VR conference. For example, the presentinvention may be utilized to replace a gaming system such as an X-Box®or a Playstation®. It is further contemplated that the present inventionmay be utilized in combination with others, wherein the multiple systemsmay be utilized for communication between each of the users. Systemssuch as this may be utilized by military forces for covertcommunications, pilots, motorsport race teams and other similar careersthat may require communication amongst more than one person. Anothercontemplated use for the present invention is warehouse management andinventory control, wherein the user may roam freely around the warehouseentering inventory, fulfilling orders, or maintaining quality control.Although the present invention has been described with regards tospecific examples these examples should not be considered limiting inany manner in that the methods of the present invention may be appliedto a wide variety of technologies, many of which were not disclosedhere.

The instant invention is shown and described herein in what isconsidered to be the most practical, and preferred embodiments. It isrecognized, however, that departures may be made there from, which arewithin the scope of the invention, and that obvious modifications willoccur to one skilled in the art upon reading this disclosure.

1. A method for controlling a microprocessor based system in a virtualenvironment, said method comprising: loading a computer program into amemory space; transmitting at least one signal from at least onetransmitter; displaying virtual input devices on a head-mounted displaydevice; wherein said virtual input devices initially have predeterminedcoordinates; receiving data generated from movement of at least onesensor; calculating x, y and z coordinates of said sensor movement;comparing calculated coordinates to the pre-determined coordinates ofeach virtual input device; calculating desired input from coordinatesgenerated by said sensor movement; and displaying input on said displaydevice and transmitting said input to said operating system; changingthe location of said virtual input devices by grabbing the virtual inputdevices in the virtual environment; and dragging the virtual devices toa new location; wherein said virtual input devices are a virtualkeyboard and a virtual pointing device.
 2. The method according to claim1, wherein said transmitter transmits at least two different signals,one signal being electromagnetic, the second being chosen from the groupconsisting of electromagnetic, optical, acoustical, and inertial.
 3. Themethod according to claim 1, wherein said sensor movement is translatedto define key presses on said virtual keyboard or movement of saidvirtual pointing device.
 4. A method for controlling a microprocessorcontrolled device in a virtual environment, said method comprising:loading a computer program into a memory space; loading an operatingsystem into a memory space; transmitting at least one signal from atransmitting device displaying a virtual keyboard and a virtual inputdevice on a display device; initializing coordinates defining individualkeys of said keyboard; initializing coordinates defining a location ofsaid input device in relation to said keyboard; receiving data at leastone sensor wherein said data received is converted into coordinateinformation and stored in a memory space; determining if coordinatesderived from movement correlate to a key location of the virtualkeyboard or movement of the virtual input device; and displaying sensormovement on said display device.
 5. The method according to claim 4,further including the step of; determining if sensor movement correlatesto movement of said virtual input device, and providing feedback to theuser to indicate movement of said virtual input device on said displaydevice.
 6. The method according to claim 5, wherein a transmission rateof said transmitter is reduced by certain percent proportion to thedistance between sensor and virtual object when it is determined that atleast one sensor moves beyond a predetermined coordinate location insaid virtual environment.
 7. The method according to claim 4, whereinsaid display device is configured to be wearable by the user.
 8. Themethod according to claim 4, wherein determining step further comprises:adding a time stamp and sensor identification tag to each data set;placing said coordinate date, time stamp data and sensor identificationdata into a memory location; comparing transmitted sensor data to theknown coordinates of the virtual keyboard and virtual input device anddetermining if the sensor date correlates to a location on the virtualkeyboard or a location on the virtual input device; and using knownthreshold values to determine if data from sensor movement correlates toa desired input chosen from the group consisting of: a key press, a keyrelease, a keyhold, movement of the virtual input device, a button presson the virtual input device or a double button press, or a buttonrelease.
 9. The method according to claim 4, further including the stepof providing said feedback in relation to sensor movement.